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Abstract. This paper explores our experiences in deploying a video application tool in film studies, 
and its evaluation in terms of realistic contextual end-users who have real tasks to perform in a real 
environment. We demonstrate our experiences and core lesson learnt in deploying our novel movie 
browser application with undergraduate and graduate students completing a Film Studies course in 
Dublin City University over a semester. We developed a system called MOVIEBROWSER2 that 
has two types of browsing modes: Advanced and Basic. In general, students found that the features 
we provided were beneficial for their studies. Some issues or mismatches arose during the trial. 
A 'wish-list’ was drawn up that might be useful for the future system developer. The contribution 
and achievements reported in this article are on the demonstration and exploration of how advances 
in technology can be deployed, and media can be accessed in the context of a real user community. 
Exploring the usage indicates a positive acceptance among students, besides lessons learned that 
are important for further investigation. 
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1. Introduction 

As a result of developments in multimedia technologies, the prospect of wide and ubiq¬ 
uitous use of new media is promising in various domains; in particular, in education 
including the use of software, hardware, Internet applications and other related activi¬ 
ties which can facilitate and improve learning. The use of video in Film Studies teaching 
and learning, for example, can bring overall usefulness and benefit, as video can be an 
important resource for students (Mohamad Ali and Smeaton, 2009). Film and media are 
more accessible and useful for a variety of purposes with the advances of information 
technology, and have played a part in the evolution of scholarly research and educational 
goals. Innovative educational technologies are possible to develop as a showcase with 
numerous potential benefits. Research towards integration of both technical multimedia 
research and user orientation plays a significant role in today’s learning. A number of ac¬ 
tivities such as forums and workshops on how to deliver multimedia research to end user 
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experience are starting to happen (i.e., QoMEX, 2009), showing a growing awareness of 
the importance of relating Multimedia research to the ‘real world’. 

Much work has been performed regarding the technical possibilities for multimedia 
content analysis, particularly video. These works usually process a video element with 
various automatic outcomes and analyses, such as shot boundary detection, image seg¬ 
mentation, and automatic detection of semantic concepts. (Mas and Fernandez, 2003; 
Smeaton, 2004; Smeaton et al., 1999). As its goal, much of the work has a focus on the 
evaluation and measurement of precision and recall for video search tasks, of the effi¬ 
ciency and accuracy of the techniques developed. User involvement, if any, will normally 
be involved only at the testing stage. Such evaluations are performed in pre-defined lab 
settings or sessions. Various applications of the new technologies can be developed as 
a showcase in order to carry out the evaluation. However, not much work has been car¬ 
ried out beyond this boundary where new technology in video analysis techniques are 
deployed and used by real end-users in real settings and with real tasks involved. User 
interactions with new software applications that incorporate these kinds of technologies 
are among the contributions demonstrated in this paper. Such true user interactions are 
almost never captured or considered. The objective of this paper is to discuss our analysis 
when exploring the usage of a video application tool as an advanced technology for real 
users in a Film Studies context. 

The paper is organized as follows. In the following section, some background work in 
video content analysis and examples of working application is introduced. This includes 
some works in deployment that combine technology into practical usage. In Section 3 
and 4, we elaborate on our developed system MOVIEBROWSER2 and explain the de¬ 
ployment trial to Film Studies student. It is followed by our findings and discussions that 
also cover some lessons learned, based on our deployment experiences and analysis. We 
end with conclusions and limitation of the work. 


2. Background Work 

A number of operational video application showcases have been developed applying the 
underlying techniques of video content analysis; particularly related to movies such as 
MovieBrowser (Lehane et al. , 2007), Videana (Ewerth et al., 2005), and a project at 
the INRIA (Ronfard, 2004). These works are among the many works that are novel and 
consider the technical perspective of the analysis (i.e., measuring Precision/Recall and 
accuracy). From another perspective, they lack assessment from the aspect of real usage 
monitoring and evaluation. Not much work has been performed which goes beyond the 
boundary of end-user orientation, where the usage of these kinds of tools or showcases 
are actually monitored and assessed. 

In addition to pure technology-driven evaluations, there is also work reported that 
combines technological efforts into a deployment stage where it is subsequently used by 
users like Ffschlar (Lee et al., 2000), Newsblaster (McKeown et al., 2002), iTV (Bern- 
haupt et al., 2007) and SportsAnno (Lanagan and Smeaton, 2007). The Ffschlar Digital 
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Video Library System was developed at Dublin City University to support capture, index¬ 
ing, browsing, searching, and summarizing of digital video, and has been deployed into 
four separate video content collections for a variety of users and application scenarios 
(Smeaton et al., 2004a). The four versions of the Ffschlar system include TV programs 
(Smeaton et al., 2004b), TV news (Smeaton et al., 2004a), TRECVid video track par¬ 
ticipation (Browne et al., 2003), and nursing educational videos (Gurrin et al, 2004). 
Information provided to users in the system interface are based on finding and selecting a 
video program either using text or metadata. Supported interface elements included a key 
frame slideshow, a hierarchical key frame browser, and a timeline browser. Ffschlar-News 
was one of the collections designed to support an archive to the main evening TV news 
broadcast. It incorporates a number of multimedia and recommendation techniques and 
was deployed within a University campus for several years, in which large scale testing 
and evaluation (performance and usability) has been carried out (Smeaton et al., 2004a). 
Methods used from video content analysis include shot boundary detection, key frame 
extraction, capture of closed captions, and the system allows for text searching, browsing 
and playback. An extended live usage study has been performed on Ffschlar-News with 
16 users using a highly qualitative and ethnographic diary (Lee et al., 2006). This study 
mainly emphasized the understanding of real use, the development of use over time, and 
the use of new technologies. 

Newsblaster at Columbia University (McKeown et al., 2002) is an experimental sys¬ 
tem incorporating natural language processing techniques to automatically crawl news 
websites on a daily basis, and summarize and present them to web users. The system has 
been deployed since 2001 and was developed mainly to demonstrate its summarization 
robustness and the use of TDT (topic detection and tracking) technology. Researchers 
on the Newsblaster system had started conducting a large online evaluation to measure 
the usage and preferences. The system has also undergone a number of experiments, 
mainly on summarization efficiency (Precision and Recall), as reported in (McKeown 
et al., 2001). 

An Austrian interactive TV (iTV) trial (Bernhaupt et al., 2007) deployed a novel TV 
application to a local cable TV provider in Salzburg, Austria, and ran for 4 months in 
2004-5. In this paper, the authors tried to overcome the lack of research in the area that 
can provide actual usage data; the researchers carried out evaluation and generated results 
from a usability test and the findings on the use of the iTV services. In order to assess ac¬ 
tual usage data, the system was deployed into more than 300 households. Apart from that, 
the design of the system also followed a user-centered design approach in order to collect 
more usable data. The researchers got low feedback from the trial due to some technical 
constraints. The system received some suggestions for improvement from the usability 
test findings; these are tackled in subsequent projects. Data collections from the server 
logs, questionnaires, and interviews were used to illustrate the findings. User acceptance 
of the new technology is considered low as this area was still immature. The developers 
followed a user-centered design in order to provide a usable system for the user, but then, 
due to time constrains, the improvements could not be carried out for the deployment 
test. The research proceeded with the improved version and was further re-deployed in 
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their future work. When trying to improve the system they stuck to a user-centered design 
approach, whereby they took into account user characteristics, experiences, expectations, 
and preferences. The user test showed that users preferred a simple navigational design. 
The research outputs also noted several issues from the trial, such as technical, usabil¬ 
ity, and user acceptance issues. For instance, during the field trial, users reported some 
technical problems which proved to be one of the reasons for low usage. They identified 
from the deployment conclusion that usability should be taken seriously to ensure user 
acceptance. 

SportsAnno is a video browsing system designed and developed to allow users to 
make comments and share opinions and ideas on soccer events with other registered 
users (Lanagan and Smeaton, 2007). SportsAnno was deployed during the soccer World 
Cup 2006. Users can browse soccer video, and at the same time, make comments and 
annotate while reading texts of newspaper reports related to the matches. Usage data 
on sports event segmentation was gathered during the deployment stage. A number of 
usage monitoring issues were pointed out from the deployment, such as low responses 
for some specific games comments. Among the solutions proposed are suggestions that 
the system should have an alert notification, which would inform or signal that there are 
new comments. Some other lessons learned were: to include direct linking and annotation 
that allow easier navigation; and locating video events within the report. By viewing 
comments of others, users will catch additional information, not available in the original 
newspaper report. 

Examples given above are some of the work in deployment efforts that combine tech¬ 
nology into practical system development. These trial efforts also indicate that there is 
growing awareness of the importance of user evaluation in realistic environments. This 
research direction will bring a new perspective into variety of research agenda in visual 
informatics that extends interaction and communication based on visual interfaces; thus, 
could support users in understanding media content. 


3. The MOVIEBROWSER2 Application 

We developed the MOVIEBROWSER2 to incorporate the use of a number of content 
analysis techniques, particularly those that identify movie scene boundaries and catego¬ 
rize them into exciting, montage, and dialogue scenes. MOVIEBROWSER2 uses several 
recent multimedia technologies to automatically process digital video content, but at the 
same time, we used a usability engineering process to relate these techniques to the real 
tasks of real users in their real environments. Our MOV1EBROWSER2 interface is shown 
in Fig. 1 (a) and (b). The designed interface incorporates some advanced features such as 
a timeline bar near the top of the screen (Fig. 1 (b)). The timeline bar will appear in three 
different colors, each representing different types of scenes. For example, the green band 
represents those scenes with Dialogues', the yellow band represents Exciting scenes; the 
pink band represents those scenes with Montage. These interactive timeline features were 
developed to provide assistance in breaking down conventional patterns of viewing and 
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Fig. 1. MOVIEBROWSER2 screen shot (a) main page, (b) main browsing movie clips. 
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helping the user to appreciate the grammar and structures of film making, and might help 
to break down the narrative and style of the film into its constituent parts. Enabling the 
film to be seen and navigated with a timeline bar outlining exactly where the sequence 
is, in relation to the whole film, might be useful when students are trying to understand 
narrative construction and the different functions of any given scene in a film. Such initial 
impressions are necessary to later help develop a more reflective as well as an intuitive 
engagement with the movie content (Brereton, P., personal communication, August 5, 
2009). 

MOVIEBROWSER2 provides the main features that, we believe, could support the 
process of reading a movie for Film Studies students. The function can be divided into 
four major categories: 

(1) Selecting a movie - list of movie posters with some other information (i.e., titles, 
genre, year) on the left side of the screen as shown in Fig. 1 (a). 

(2) Browsing within-movie - the user can browse using the timeline bar on the upper 
side of the screen and using a key frame view as shown in Fig. 1 (b). Each timeline 
represents a segmented scene of dialogue, exciting, or montage that is identified in 
each movie. 

(3) Playing a movie - standard buttons as can be seen in the normal media player 
were provided in MOVIEBROWSER2; namely, play, pause, stop, full screen, vol¬ 
ume adjustments and mute for the user to play movie clips. The playback area in 
MOVIEBROWSER2 can be seen in Fig. 1 (b) at the right side of the interface. 

(4) Social interactions - we provide a feature so that, at any point during the playback, 
the user can make notes by clicking on the ‘Make Some Notes/Comments’ button 
underneath the playback area, and then type a note on an interesting scene or the 
whole movie; which, perhaps, can be saved and viewed later and even shared with 
other students (Fig. 1 (b)). 


4. Deployment Trial Methodology 

We deployed our MOVIEBROWSER2 system in groups of Film Studies students in 
School of Communications, Dublin City University. Students used the conventional ap¬ 
proach, whereby they got DVDs from the University library or rental shop and used these 
as the basis for readings of movies as part of course assignments. The main restrictions 
contributing to the problems in their studies were, getting short-term loans, and a lack 
of DVD resources (i.e., old production year, obsolete format). In preparing to write a 
textual analysis for their course, students of Film Studies are given a list of topics to be 
chosen from prior to that. Once decided on the particular topic, students start looking 
for related DVD movies. The length of the textual analysis varies across modules from 
1,000 to 3,000 words, which depends on the level of the module. There is no unique for¬ 
mula to decode a film. A textual analysis must engage with the grammar of the cinema 
and the most important thing is to concentrate on how the meaning is created (Brere¬ 
ton, 2008a; Brereton, 2008b). The typical tasks of Film Studies students are to read and 
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analyze movie sequences. Reading a movie in their context refers to the process of un¬ 
derstanding and analyzing movie content closely, looking for different levels of meaning 
and critique for example from elements like framing, depth of field, plot, shots, camera 
angle, lighting and so on. On a broader level it also involves an understanding of the 
generic conventions and narrative structure of individual movies (Brereton, 2008a). Our 
current design of browsing movie content could also provide a positive impact and will 
aid in film analysis and could provide an indirect process of reading a film. In relation 
to the MOVIEBROWSER2 system we developed, we believe that this new strategy and 
‘tool’ is also useful to help students to acquire the skills of reading film and there are 
some elements or interface features that could maximize the potential of their learning. 
For example, enabling the film to be seen and navigated with a timeline bar, outlining 
exactly where the sequence is, in relation to the whole film, for example, might be very 
useful when students are trying to grapple with narrative construction and the different 
functions of any given scene in a film. 

4.1. Participants 

Our group of real users at the deployment stage were students from the modules CM272 
National and Ireland Cinema, which is a second year undergraduate level course of one 
semester (12 weeks classes), and CM135 Analyzing Media Content , a first year under¬ 
graduate level module, also of one semester duration. The combined class groups totaled 
268 students (CM272 = 76 students, CM135 = 192 students). For the former module, the 
lecturer used mostly an Irish cinema maker as their main focus, while the latter module 
used contemporary Hollywood movies. Students taking the modules were from differ¬ 
ent degree programmes including the BA in communications, the BSc in multimedia and 
the BA in journalism. In both modules, the students’ main task as part of their practical 
assessment is to “read” movies and analyze the film in detail based on what they have 
learned in class (i.e., wide range theories, historical context of cinema and etc) and from 
this “reading” they are required to produce an essay on a certain topic. There is no written 
examination at the end of either module. Student assessments are based on the essays they 
produce from reading the movies. For instance, in the CM135 module, students need to 
produce an essay on any chosen movie, of approximately 1000 words in length. This task 
involves an individual reading of a sequence of any chosen movie. Each student needs 
to analyze a movie from various aspects such as grammar of the cinema, mise-en-scene 
(what is in the frame), capturing the essence in the movie, focusing on aspects like music, 
camera, lighting, and others. 

4.2. Tasks 

As part of our end-user involvement effort, we deployed our application for managing and 
browsing movie contents to film study students (taking modules CM272 and CM135) for 
the duration of the whole spring semester. We provide two types of browsing features in 
our tool; Basic and Advanced interface. The advanced type of browsing (Fig. 2 (b)) has 
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(a) (b) 

Fig. 2. Example of screenshot page (a) basic interface (b) advanced interface. 


features that could enhance user browsing, like the inclusion of a visual timeline, sup¬ 
port for key frame browsing, note taking features. On the contrary, some of the movies 
were listed under the basic category interface-type of browsing that provides only a ba¬ 
sic playback function with normal standard DVD-like player interface facilities such as 
play, pause or slider bar (Fig. 2 (a)). The collection of 30 movies came from various gen¬ 
res (i.e., comedy, drama, romance, and action), ranging from contemporary Hollywood 
movies to old Irish movies, with production years from 1952 to 2004 as listed in Table 1. 


Table 1 

List of Movies in MOVIEBROWSER2. 


With advanced features 

With basic features 

Shrek (2001) 

American Beauty (1999) 

About Adam (2000)* ** 

The Most Fertile Man In Ireland (1999)* 

Circle of Friends (1995)* 

High Fidelity (2000) 

Intermission (2003)* 

Oceans 11 (2001) 

Into The West (1992)* 

The Magdelene Sisters (2002)* 

Man About Dog (2004)* 

Lilo and Stitch (2002) 

Michael Collins (1996)* 

Road To Perdition (2002) 

On The Edge (2001)* 

This Is My Father (1998)* 

Spin The Bottle (2003)* 

The Royal Tenenbaums (2001) 

The Quiet Man (1952)* 

Eat The Peach (1986)* 

The Butcher Boy (1991)* 

Poitin (1979)* ** 

Korea (1995)* ** 

After 68 (1993)* ** 

Nora (2000)* ** 

Goldfish Memory (2003)* ** 

- 

The Shipper (1993)* ** 

- 

The Visit (1992)* ** 

- 

The Ballroom of Romance (1982)* ** 

- 

Bent Out of Shape (1995)* ** 


Note: * Irish-directed movies; ** movie used in CM272 module. 

















Exploring the Usage of a Video Application Tool: Experiences in Film Studies 


171 


There were a few movies that were short in duration (less than 1-hour) that were used 
in class such as After 68 [25 minutes], Bent Out Of Shape [27 minutes], The Visit [19 
minutes] and The Ballroom of Romance [50 minutes]. These movies were categorized 
under the standard browsing features due to their short lengths and difficulties in gen¬ 
erating event detections and classifications as a result of that short length. We tried to 
balance the number of movies that were categorized into ‘Advanced’ or ‘Basic’ interface. 
Movies with (**) are examples of Irish-directed movies that are also used as part of the 
discussions in the CM272 module. 

4.3. Procedures 

We provided a system demo in the earlier part of the semester for each class. The lectur¬ 
ers for each module were acknowledged and informed of the whole procedure before any 
deployment was carried out; they gave great support and encouragement for the process 
to proceed. A brief explanation was given to each class regarding the usage of the system, 
during a demo presentation of about 15 minutes, after which an email was sent to all the 
students, informing them of their individual system username and password, along with 
information on the system (i.e., web links, player installation and other requirements). 
Follow-up email reminders were sent a few times during the period of deployment. We 
administered an online questionnaire in week-13 and week-14 of this initial deployment. 
Students’ usage data logs were captured automatically. At the end of the deployment, an 
email was sent to students to thank them and acknowledge their responses and coopera¬ 
tion. 

4.4. Data Capture 

We separated the captured data as follows: 

• Usage Logs: Our objective in performing the deployment on the students in the 
University was to monitor and assess their actual-usage data of the developed 
movie browsing and playback tool. Among the logs captured were the interactions 
or user actions with the features that we provided on the screen interfaces, either 
Basic or Advanced. 

• Questionnaires: The objectives of the administered online questionnaires was to 
measure subjective satisfaction and overall acceptance with the new deployed sys¬ 
tem, and to collect quantitative and qualitative data on the system we deployed. 
This included questions on the demographics on students’ gender, age, course in¬ 
formation, and other preliminary inquiries; as well as their overall reaction to the 
system in terms of perceived satisfaction, features provided, and opinions on the 
value of technology incorporated. 


5. Findings 

Our findings from our various types of logging and qualitative assessments are divided 
into users, and user interface features, which we now present. 
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5.1. Users 

Out of a total of 268 students in both classes, 107 students (40%) accessed the 
MOVIEBROWSER2 at least 2 times during the deployment period , which tells us that 
they were not all wholly engaged with the system. 17 users (16%) each accessed the 
system for up to 8 hours in total, most over several sessions. As shown in Table 2, the 
total access duration time was approximately 86 hours (CM272 = 57 hours, CM135 = 
29 hours). All 7 Hollywood movies that were stored in the system library were accessed 
a total of 73 of times (39%), with Shrek (2001) being mostly accessed and viewed by 
students (Fig. 3). At first glance this may not seem substantial, but deeper analysis shows 
that the access was very focused on the students’ tasks, thus was a time-saving service. 


Table 2 

Advanced and Basic page hours spent 


Type Hours spent 

Advanced screen 45 

Basic screen 33 

Total playback activities (Advanced and Basic) 78 
Total accessed hours in the semester 86 

Other activities* 8 


* Other than playback activities. 



Movies 


Fig. 3. Advanced vs. Basic viewed movies. 
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5.2. Interface Features 

We divided our movie collection into ‘ Advanced’ and ‘Basic’ to see the pattern of user 
interactions when extra features were incorporated. The advanced browser has features 
mainly designed to enhance the movie reading based on the three event categorizations 
(i.e., montage, dialogue and exciting). Extra interactions were found, the extra features 
including a timeline visual display, note taking, and the shot key frame view. There is 
an indication that users are engaged more on the advanced page, and this result is also 
reflected by the increased hours spent on the advanced interface as can be seen from 
Table 2, with 45 hours spent as compared to 33 hours in the standard or basic interactions. 

5.3. Usage Over Time 

Figure 4 illustrates group usage over the full semester deployment. Students started using 
the system from the 10th of March. The system had quite heavy usage approaching the 
deadline for the assignment on May 9th for both modules. It is believed that students 
tend to concentrate on their assignment just before the deadline. Increased usage was 
found during the month of April and into the early weeks of May. These patterns strongly 
suggest that students were engaged with assignments from other modules and had to 
follow their own priority deadlines. 

We also observed some usage by students after the trial deployment and assignment 
deadline, but that is not shown in Fig. 4, as they still had access to MOVIEBROWSER2 
at this time. This pattern indicates positive acceptance of the system, but we cannot be 
certain of this without engaging with the students more deeply through interviews and 
questionnaires, which we now report on. 
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Fig. 4. Students’ usage in the semester. 
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5.4. User Opinions 

60 students (56%) responded to questionnaires that we administered during weeks 13 
and 14 of the semester, using a scale from 1 to 5 (1 = Totally Disagree; and 5 = Totally 
Agree). At least 50% of participants gave a positive opinion on overall reactions to the 
system; either ‘Totally Agree’ or ‘Agree’; while negative opinion does not exceed 10% of 
the total number of responses. In general, we obtained above average values with mode 
and median equal to 4 (Agree) for all differential statements. 

In addition to quantitative evaluation, we sought more qualitative feedback from users 
through interviews, and overall comments given by students affirm the survey ratings. The 
comments include the following extracts: 

• “I love it! ” 

• “Easy to understand and use”. 

• “Yes, I seem to watch more movies with the interest of study in mind (i.e., looking 
at camera angles, edits etc) which I never did before having the browser”. 

We captured students’ opinions on their learning perception as related to their studies, 
and how the MOVIEBROWSER2 tool might support their learning. Statement 1 in the 
questionnaire was, “The features provided in the tool were useful in my studies”. This 
statement gets the highest positive response as compared to the rest of the statements 
(68% of students totally agree or agree), although 13 students (22%) gave “Neutral” re¬ 
sponses. Statement 2, “The movie collection is useful in my studies”, draws out nearly half 
participants to totally agree or agree (47%), and 20% (12 participants) gave the opposite 
opinion. Statement 3, “The too! helped me to support my information needs”, elicited 
52% of responses as total agreement. Statement 4, “The tool helped me to enhance my 
ability to read films well”, and statement 5, “The tool helped me to discover new ideas in 
my work ”, elicited 43% and 37% each, for totally agree or agree opinions. These state¬ 
ments are more abstract and related to the learning ability of individual students and their 
skills, which seems difficult to justify. However, we received positive comments such 
as, “ Helped me look for styles used in film, i.e., camera cuts, shots etc”, and, “The tool 
helped me to choose what film to use for my assignment The last two statements in 
the questionnaire, “The tool helped me to explore the collections better”, and, “The tool 
helped me to analyze the scenes better than a DVD”, draw out more than half of opinions 
in total agreement (53% and 58% each). 

To identify the features or system functions that students liked or disliked, we gave 
students opportunities to provide qualitative feedback on each. Table 3 summarizes sev¬ 
eral items mentioned most frequently by users of the system - both likes and dislikes, 
based on questionnaire responses. From the table, it can be seen that note-taking was 
among the most frequently mentioned features (likes) as 21 (30%) out of the total 
(71 mentioned), followed by event categorization (24%), timeline (18%), key frame view 
(11%), and playback of the movie (8%). 

The facility to take notes while playing a movie scene seems to be an obviously ad¬ 
vantageous function. The timeline visual and key frame view which highlight where the 
action, dialogue, and montage scenes overlap in a movie were praised as very useful, 
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Table 3 

Frequency of mentioned system features 


Features 

System-likes 

System-dislikes 

Note-taking 

21 

i 

Event categorization 

17 

- 

Visual timeline 

13 

i 

Keyframe view 

8 

i 

Movie playback 

6 

- 

Other: 

System design 

4 

3 

Access 

2* 

3** 

Limited movies 

- 

10 

Streaming problem 

- 

9 

Compatibility 

- 

3 

Total Mentioned 

71 

31 


‘convenience, “less coverage. 


indicating that a strong temporal orientation with additional cues on movie content is 
useful, as some comments below illustrate: 

• “The timeline feature was probably the most useful feature on the browser”. 

• “The timeline as it breaks down the film into the various sections - montage, action, 
dialogue, etc. - this makes it easier to carry out a more in depth analysis of the 
movie ”, 

• “Timeline. Much easier to navigate through a film”. 

The first five features in Table 3 reveal the results on interface features. We noticed 
some other positive features from reading participants’ comments which we categorize 
under “system design and access”. For example, in the system design, comments were 
mainly on the clear presentation layout which makes it easy to navigate. We also col¬ 
lected feedback on what matters distracted our users (did not like). There are very few 
comments related to system features as there was only one mention found each for the 
timeline, notes, and key frame view aspects. We also noticed an accessibility problem as 
the most-mentioned issue in the ‘system-dislikes’ column, namely the limited number of 
movies that were stored in the library which might restrict usage. Other comments we 
read were such as system compatibility (i.e., MAC user/Internet browsers). The trial ver¬ 
sion of MOVIEBROWSER2 was only compatible with Internet Explorer and Windows. 
Ease of access in the ‘system-likes’ column, meant, as a convenience factor by users; 
but in the ‘system-dislikes’ column, it meant, less coverage of access. When comparing 
the frequency of mentioned items which were either likes or dislikes, we noticed that no 
issues arose much on the “system design” and “features provided” aspects. Most of the 
system dislikes were related to the system accessibility issue. 

We also asked our users about their overall experiences in using MOVIEBROWSER2 
after the semester was completed. Among these 60 students, 43 of them (72%) said they 
would use it in future. We calculated the positive and negative expressions of their over- 
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all experiences, and we estimate that 19 of them (32%) gave positive expressions and 
only 4 (7%) gave a negative tone of expression, while the rest 37 (62%) did not express 
either positive or negative expressions. Examples of positive expressions include “I’m 
very happy/discovered ... ” and “I found it is useful/able to ... ”, and negative such as 
“Not enough/database is too small... ”. “Not enough movie selections or small database” 
refers to only thirty movies that are available to be accessed. Some of the other user 
comments are repeated below: 

• “Very happy with the system overall, I think 1 would like to use it more in the future 
as I found it useful ”. 

• “The tool made my work much quicker. It was generally very helpful. Sort of like 
an upgrade. Metaphorically speaking, I am now driving, while I had just been 
cycling ”. 

• “Brilliant system! Hopefully in the future it will work on MACs and on different 
Internet browsers! ” 

• “I was able to complete my assignment using some of the skills I had required 
on this site. The tools provided for reading a film were new to me and my work 
benefited greatly from these ”. 

Finally, we sought feedback from students on their wishlists of features for future 
versions. Some of these appear due to the difficulties in the implementation and would 
not be expected during the development design stage. The list of entries was classified 
into the following: 

• Larger and varied type of movie database (i.e., Irish, Hollywood or Europe). 

• System compatibility (i.e., other browsers and MAC users). 

• Improved accessibility (i.e., off-campus). 

• Technical constraints (i.e., high-speed access). 


6. Discussion 

Our findings have raised several issues for discussion which we now address, including 
users, features, the mismatch between these, and the lessons we learned. 

6.1. Users 

The results from our deployment established the usage pattern on how the students of 
Film Studies used the newly-introduced application that adopted advances in multimedia 
technologies. Out of 268 students, 107 (40%) used the software application for over 86 
hours during the trial. Some students would have preferred to access such material from 
off-campus, and given the ease with which video can be streamed from services such as 
YouTube and Netflix, there is an expectation among users for this same level of availabil¬ 
ity from all video services. We consider how these limitations would have affected usage. 
This may be the reason why 60% of the students did not use the system. 
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6.2. Interface Features 

The time spent by students on the ‘ Advanced’ page that provides extra features for an en¬ 
hanced type of browsing experience was much longer than the ‘Basic ’ page that only can 
play the movie using standard navigational buttons. In addition, the captured usage logs 
show much greater levels of interaction with the advanced system. This suggests students 
are more engaged with the afforded features. Positive comments on these features such 
as timeline, key frame browsing and event categorization, support this hypothesis. Users 
jumped from one point in the movie to another point easily in the movie, using visual 
representations of a timeline or the shot key frame view. Instead of playing sequences 
from the normal playback interaction, for example, either using the pause button or slider 
bar, the playback of sequences shifted to playing from the shot key frame view. These 
artificial divisions were found to influence usage among the students as provided in the 
Advanced system. 

However, just because there are more clicks and greater system interaction does not 
necessarily mean that there is better usage or enhanced user experience. It might mean 
that it is more complicated to use or features new things, thus users need to explore 
more to find out what they are looking for. The findings in the qualitative comments 
given by students reveal a list of system-likes for these extra features as well as some 
complaints from students about having no advanced features for some movies. With these 
comments, we can deduce that the greater amount of interaction and longer time spent is 
not perceived as a negative during the deployment. 

The design interface might influence the “play” element, in which movies were cat¬ 
egorized into event types with a timeline and a key frame shot view. Thus it will affect 
the users’ perceived enjoyment in using the system, where using it becomes less like 
work and more like play. We relate this phenomenon to the flow theory in work and 
organizational psychology, identified by Csikszentmihalyi (1991, 1997) where someone 
can experience flow when the technology being used actually eases the burden of their 
physical effort. This is one of the many ways of achieving flow which is evidenced here 
from users’ positive expressions and overall experiences in using the software as reported 
earlier. Measuring flow (timelessness) or play elements in a software application that in¬ 
corporates novel technology could be an interesting future research path. In this paper, 
we only provide the rationale for this phenomenon in interpreting our findings. 

6.3. Dealing with Mismatch 

In response to early requirements analysis, we incorporated a note-taking feature into 
MOVIEBROWSER2. Being able to write down comments or ideas at any point dur¬ 
ing a movie was identified as an important and useful feature from the very beginning, 
when initial student needs were captured. However, the feature was underused during the 
deployment period, as according to the log interaction data, it was only used by three stu¬ 
dents. This shows a mismatch between what our users said would be beneficial and what 
they actually used in practice. Interestingly, in the post-trial questionnaire, we got very 
positive responses regarding the benefit of this feature, even though they did not actually 
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use it. Two of the three participants who actually used the notes feature commented very 
positively about the feature in their post-trial questionnaire (the third participant did not 
respond to the questionnaire). 

• “Note-taking, it helps you keep track of information you are taking down on a 
particular part of the movie and helps you remember”. 

• “I liked the note-taking the best as I could take notes quickly and easily during 
a sequence”. 

• “Note taking section is very useful”. 

• “The way that events are organized, the ability to take notes, all movies on the 
course were available and they are not readily available elsewhere”. 

After deployment, we performed a qualitative investigation as to why they did not use 
this feature during deployment, and received feedback from 15 participants. The reasons 
for not using the notes feature during the trial were analyzed and grouped into categories 
summarized as following: 

• Privacy issue: “Idon't want my colleagues to steal my ideas'.” 

• Preference for conventional practice: “I like to write with pen and paper!”. 

• Access limitation: 1/ wanted to do it at home!” 

6.4. Lessons Learned 

From our experiences and analysis we formulated a number of lessons learned which 
include things that were not expected, and for each of these we have a number of sugges¬ 
tions. 

• The mismatch between user requirement and actual use - we followed the usability 
engineering method from the very beginning of the application development that 
included a user requirements stage which took place before the system was built. 
We conducted interviews and observations, but in the system deployment stage 
we found that note-taking was underused. This shows a mismatch between the 
needs we captured and the actual practices from the user. In the system testing for 
the developed system, we did not foresee this situation as users always gave good 
feedback on the features we provided. Furthermore, they were not really attached 
to any usage of the software application as it was only for testing purposes before 
we underwent a real deployment stage which identified the mismatch. This tells 
us that we should be more understanding and aware of the usage context when 
designing such applications. Further system design should emphasize the context 
of usage even more, rather than simply the functions that the features can provide, 

• Accessibility factors - the features that were provided in the application did not 
raise many issues during the trial, compared to the benefits of accessibility. The 
accessibility of the application was clearly an important criterion which led to high 
usage. An improvement in the system accessibility would increase the usage among 
users even more so. 

• Expert point of view - from the academic point of view there is always an inher¬ 
ent difficulty reading a movie as it is a very subjective experience. The academic 
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teaching processes attempt to make this as objective and scientific as possible, so 
that students can learn the skill of writing an analysis of film using robust strate¬ 
gies. This has been an agenda item for a long time now, in the domain of Film 
Studies. We discovered that developing an end-to-end system that leverages cur¬ 
rent research, and deploying this into a real usage situation (Film Studies context), 
is difficult, because some of the technology elements are immature and can easily 
hinder the usage and perceived value of the system. 


7. Conclusions 

The paper presents our findings when exploring the usage of a video browsing tool which 
was made available to students of Film Studies as part of their University curriculum. 
The results presented highlight some interesting findings for students of Film Studies in 
browsing and playing movie content. User access and usage were found to be varied and 
influenced by many factors. In general, students found the features we provided were 
beneficial for their studies. Some issues or mismatches arose during the trial. A ‘wish- 
list’ was drawn up that might be useful for the future system development. Our biggest 
strength in this study was the fact that the interactions we logged and observed were 
from real users, students of the School of Communications at Dublin City University. 
Our deployment was a longitudinal study for a real module over a whole semester, and 
was found to be a useful (although time-consuming) exploration process. A mismatch be¬ 
tween students’ initial wishes for a ‘note/comments’ feature and its actual usage during 
the trial triggered more questions for us, of how users’ wishes collected at the require¬ 
ments engineering stage should be interpreted in the context of usage, rather than treated 
as an isolated feature in itself. 

This work provides insights into how a technologically-oriented research field, nor¬ 
mally driven by technical perspectives of novel multimedia analysis, could be aligned to 
a real-world setting with real users who have real information needs. Bridging the gap 
between real usage with and technical possibilities was described in this paper, and the 
novelty factor is that we filled the gap of assessing real usage from the deployment and 
monitoring process into real end users and their real contextual environments. Little pre¬ 
vious work has been reported in this area, logging and interpreting user interactions with 
new software applications that incorporate new kinds of technologies in a real, opera¬ 
tional setting. 

This study, however, also has its limitations. In this trial deployment, we designed and 
deployed a tool that has some technical constraints, thus we could not get optimum usage 
from our users. Future work should consider further improvement in the application itself 
as well as more users, and should be carried out over a longer period of time. 
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Taikomuju vaizdo priemoniii tyrinejimas: filmu kurimo patirtis 

Nazlena Mohamad ALI, Alan F. SMEATON 

Straipsnyje aptariama keletas taikomuju vaizdo priemoniu. kurias studentai paskaitu metu nau- 
dojo Dublino universitete. Studentu duomenys buvo fiksuojami ir analizuojami. Tyrime dalyvavo 
268 studentai, kurie turejo atlikti tam tikras uzduotis. Skatintinas kino filmu konteksto analiza- 
vimas. kuris apima giliu turinio bei specialiuju efektu apzvalgu. Techninis daugialypes jrangos 
tyrinejimas bei gera vartotojo orientacija daro didel? jtaka mokymuisi. 



